In [5]:
# drop unlabled data in train/test dataset
df = df[df[target_col].notna()]
sns.displot(x=target_col, data=df, kind='kde') # kernel density estimate
Out[5]:
<seaborn.axisgrid.FacetGrid at 0x7f03a0b4ae80>
by the Oracle AutoMLx Team
AutoMLx Regression Demo version 23.1.1.
Copyright (c) 2023 Oracle, Inc.
Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
In this notebook we will build a regressor using the Oracle AutoMLx tool for the public California Housing dataset to predict the value of house prices. We explore the various options provided by the Oracle AutoMLx tool, allowing the user to control the AutoML training process. We finally evaluate the different models trained by AutoML. Depending on the dataset size and the machine running it, it can take about tens of minutes. The dataset is sampled down for a snappier demo, with the option to run it with the full dataset. We finally provide an overview of the possibilites that Oracle AutoMLx provides for explaining the predictions of the tuned model.
Data analytics and modeling problems using Machine Learning (ML) are becoming popular and often rely on data science expertise to build accurate ML models. Such modeling tasks primarily involve the following steps:
All of these steps are significantly time consuming and heavily rely on data scientist expertise. Unfortunately, to make this problem harder, the best feature subset, model, and hyperparameter choice widely varies with the dataset and the prediction task. Hence, there is no one-size-fits-all solution to achieve reasonably good model performance. Using a simple Python API, AutoML can quickly (faster) jump-start the datascience process with an accurately-tuned model and appropriate features for a given prediction task.
! pip install seaborn==0.12.1
%matplotlib inline
%load_ext autoreload
%autoreload 2
Requirement already satisfied: seaborn==0.12.1 in /scratch_user/ypushak/automl-3/automl/py_3.8.7/lib/python3.8/site-packages (0.12.1) Requirement already satisfied: matplotlib!=3.6.1,>=3.1 in /scratch_user/ypushak/automl-3/automl/py_3.8.7/lib/python3.8/site-packages (from seaborn==0.12.1) (3.5.1) Requirement already satisfied: pandas>=0.25 in /scratch_user/ypushak/automl-3/automl/py_3.8.7/lib/python3.8/site-packages (from seaborn==0.12.1) (1.4.1) Requirement already satisfied: numpy>=1.17 in /scratch_user/ypushak/automl-3/automl/py_3.8.7/lib/python3.8/site-packages (from seaborn==0.12.1) (1.22.2) Requirement already satisfied: fonttools>=4.22.0 in /scratch_user/ypushak/automl-3/automl/py_3.8.7/lib/python3.8/site-packages (from matplotlib!=3.6.1,>=3.1->seaborn==0.12.1) (4.29.1) Requirement already satisfied: pillow>=6.2.0 in /scratch_user/ypushak/automl-3/automl/py_3.8.7/lib/python3.8/site-packages (from matplotlib!=3.6.1,>=3.1->seaborn==0.12.1) (9.0.1) Requirement already satisfied: packaging>=20.0 in /scratch_user/ypushak/automl-3/automl/py_3.8.7/lib/python3.8/site-packages (from matplotlib!=3.6.1,>=3.1->seaborn==0.12.1) (21.3) Requirement already satisfied: pyparsing>=2.2.1 in /scratch_user/ypushak/automl-3/automl/py_3.8.7/lib/python3.8/site-packages (from matplotlib!=3.6.1,>=3.1->seaborn==0.12.1) (2.4.7) Requirement already satisfied: python-dateutil>=2.7 in /scratch_user/ypushak/automl-3/automl/py_3.8.7/lib/python3.8/site-packages (from matplotlib!=3.6.1,>=3.1->seaborn==0.12.1) (2.8.2) Requirement already satisfied: kiwisolver>=1.0.1 in /scratch_user/ypushak/automl-3/automl/py_3.8.7/lib/python3.8/site-packages (from matplotlib!=3.6.1,>=3.1->seaborn==0.12.1) (1.1.0) Requirement already satisfied: cycler>=0.10 in /scratch_user/ypushak/automl-3/automl/py_3.8.7/lib/python3.8/site-packages (from matplotlib!=3.6.1,>=3.1->seaborn==0.12.1) (0.10.0) Requirement already satisfied: pytz>=2020.1 in /scratch_user/ypushak/automl-3/automl/py_3.8.7/lib/python3.8/site-packages (from pandas>=0.25->seaborn==0.12.1) (2022.1) Requirement already satisfied: six in /scratch_user/ypushak/automl-3/automl/py_3.8.7/lib/python3.8/site-packages (from cycler>=0.10->matplotlib!=3.6.1,>=3.1->seaborn==0.12.1) (1.16.0) Requirement already satisfied: setuptools in /scratch_user/ypushak/automl-3/automl/py_3.8.7/lib/python3.8/site-packages (from kiwisolver>=1.0.1->matplotlib!=3.6.1,>=3.1->seaborn==0.12.1) (62.3.4) [notice] A new release of pip available: 22.2.2 -> 23.0 [notice] To update, run: pip install --upgrade pip
Load the required modules.
import gzip
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import fetch_california_housing
from sklearn.metrics import mean_squared_error
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
# Settings for plots
plt.rcParams['figure.figsize'] = [10, 7]
plt.rcParams['font.size'] = 15
sns.set(color_codes=True)
sns.set(font_scale=1.5)
sns.set_palette("bright")
sns.set_style("whitegrid")
import automl
from automl import init
Dataset details are available here: https://scikit-learn.org/stable/datasets/real_world.html#california-housing-dataset. The goal is to predict the median price of a house given some features.
X, y = fetch_california_housing(return_X_y=True)
ds = fetch_california_housing(return_X_y=False)
df = pd.concat([pd.DataFrame(X, columns=ds.feature_names),
pd.DataFrame(y.ravel(), columns=['Median Price'])], axis=1)
target_col='Median Price'
df.shape
(20640, 9)
df.head()
| MedInc | HouseAge | AveRooms | AveBedrms | Population | AveOccup | Latitude | Longitude | Median Price | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | 8.3252 | 41.0 | 6.984127 | 1.023810 | 322.0 | 2.555556 | 37.88 | -122.23 | 4.526 |
| 1 | 8.3014 | 21.0 | 6.238137 | 0.971880 | 2401.0 | 2.109842 | 37.86 | -122.22 | 3.585 |
| 2 | 7.2574 | 52.0 | 8.288136 | 1.073446 | 496.0 | 2.802260 | 37.85 | -122.24 | 3.521 |
| 3 | 5.6431 | 52.0 | 5.817352 | 1.073059 | 558.0 | 2.547945 | 37.85 | -122.25 | 3.413 |
| 4 | 3.8462 | 52.0 | 6.281853 | 1.081081 | 565.0 | 2.181467 | 37.85 | -122.25 | 3.422 |
We first display the density of the target median price.
# drop unlabled data in train/test dataset
df = df[df[target_col].notna()]
sns.displot(x=target_col, data=df, kind='kde') # kernel density estimate
<seaborn.axisgrid.FacetGrid at 0x7f03a0b4ae80>
We now separate the targets (y) from the training features (X) and split the dataset into training (70%) and test (30%) datasets. The training set will be used to create a Machine Learning model using Oracle AutoMLx, and the test set will be used to evaluate the model's performance on unseen data.
X_full = df.drop(target_col, axis=1)
y_full = df[target_col]
X_train, X_test, y_train, y_test = train_test_split(X_full, y_full, test_size=0.3, random_state=7)
X_train.shape, X_test.shape
((14448, 8), (6192, 8))
The AutoML pipeline offers the function init, which allows to initialize the parallel engine. By default, the AutoML pipeline uses the dask parallel engine. One can also set the engine to local, which uses python's multiprocessing library for parallelism instead.
init(engine='local')
[2023-02-10 12:50:27,097] [automl.xengine] Local ProcessPool execution (n_jobs=40)
The Oracle AutoMLx solution provides a pipeline that automatically finds a tuned model given a prediction task and a training dataset. In particular it allows to find a tuned model for any supervised prediction task, e.g. classification or regression, where the target can be binary, categorical or real-valued.
AutoML consists of five main modules:
All these pieces are readily combined into a simple AutoML pipeline which automates the entire Machine Learning process with minimal user input/interaction.
The AutoML API is quite simple to work with. We create an instance of AutoML pipeline. Next, the training data is passed to the fit() function which successively executes the previously mentioned modules.
est1 = automl.Pipeline(task='regression')
est1.fit(X_train, y_train)
[2023-02-10 12:50:27,489] [automl.pipeline] Random state (7) is used for model builds
[2023-02-10 12:50:27,527] [automl.pipeline] Dataset shape: (14448, 8)
[2023-02-10 12:50:27,630] [automl.pipeline] Running Auto-Preprocessing
[2023-02-10 12:50:27,661] [automl.pipeline] Preprocessing completed. Updated Dataset shape: (14448, 8), cv: [(11558, 2890), (11558, 2890), (11558, 2890), (11559, 2889), (11559, 2889), (11558, 2890), (11558, 2890), (11558, 2890), (11559, 2889), (11559, 2889)]
[2023-02-10 12:50:27,857] [automl.pipeline] SVR, KNeighborsRegressor are disabled for datasets with > 10K samples or > 1K features
[2023-02-10 12:50:27,858] [automl.pipeline] Running Model Selection
[2023-02-10 12:50:39,019] [automl.pipeline] Model Selection completed. Selected model: ['LGBMRegressor']
[2023-02-10 12:50:39,021] [automl.pipeline] Running Adaptive Sampling. Dataset Shape: (14448, 8), Valid Shape: None, CV: [(11558, 2890), (11558, 2890), (11558, 2890), (11559, 2889), (11559, 2889), (11558, 2890), (11558, 2890), (11558, 2890), (11559, 2889), (11559, 2889)], Class counts: N/A
[2023-02-10 12:50:44,643] [automl.pipeline] Adaptive Sampling Completed. Updated Dataset Shape: (14448, 8), Valid Shape: None, CV: [(11558, 2890), (11558, 2890), (11558, 2890), (11559, 2889), (11559, 2889), (11558, 2890), (11558, 2890), (11558, 2890), (11559, 2889), (11559, 2889)], Class counts: N/A
[2023-02-10 12:50:44,644] [automl.pipeline] Starting Feature Selection 0. Dataset Shape: (14448, 8)
[2023-02-10 12:50:49,531] [automl.pipeline] Feature Selection 0 completed. Updated Dataset shape: (14448, 6)
[2023-02-10 12:50:49,613] [automl.pipeline] Tuning LGBMRegressor
[2023-02-10 12:51:15,445] [automl.pipeline] Tuning completed. Best params: {'boosting_type': 'gbdt', 'colsample_bytree': 0.7952797110155084, 'max_depth': 63, 'n_estimators': 377, 'num_leaves': 31, 'reg_alpha': 0, 'reg_lambda': 0, 'subsample': 1}
[2023-02-10 12:51:15,508] [automl.pipeline] (Re)fitting Pipeline
[2023-02-10 12:51:16,524] [automl.xengine] Local ProcessPool execution (n_jobs=40)
[2023-02-10 12:51:16,613] [automl.pipeline] AutoML completed. Time taken - 48.634 sec
Pipeline(model_list=['AdaBoostRegressor', 'DecisionTreeRegressor',
'ExtraTreesRegressor', 'LGBMRegressor', 'LinearSVR',
'LinearRegression', 'RandomForestRegressor', 'SVR',
'XGBRegressor', 'TorchMLPRegressor'])In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. Pipeline(model_list=['AdaBoostRegressor', 'DecisionTreeRegressor',
'ExtraTreesRegressor', 'LGBMRegressor', 'LinearSVR',
'LinearRegression', 'RandomForestRegressor', 'SVR',
'XGBRegressor', 'TorchMLPRegressor'])A model is then generated (est1) and can be used for prediction tasks. We use the mean_squared_error scoring metric to evaluate the performance of this model on unseen data (X_test).
y_pred = est1.predict(X_test)
score_default = mean_squared_error(y_test, y_pred)
print(f'Mean squared error on test data : {score_default}')
Mean squared error on test data : 0.19959317236382942
During the Oracle AutoMLx process, a summary of the optimization process is logged, containing:
AutoML provides a print_summary API to output all the different trials performed.
est1.print_summary()
| Training Dataset size | (14448, 8) |
| Validation Dataset size | None |
| CV | [([0, 1, 3, 4, 6, 7, 9, 10, 13, 14, 16, 17, 18, 19, 20, 22, 23, 24, 25, 26, 27, 29, 30, 31, 32, 33, 34, 38, 39, 40, 41, 42, 43, 44, 45, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 64, 68, 70, 71, 72, 73, 74, 75, 76, 77, 78, 80, 82, 83, 84, 86, 87, 88, 90, 92, 93, 95, 96, 97, 98, 99, 102, 103, 104, 105, 106, 107, 110, 111, 112, 113, 114, 118, 119, 120, 122, 124, 126, 127, 128, 129, 130, 131, 132, 133, ...], [2, 5, 8, 11, 12, 15, 21, 28, 35, 36, 37, 46, 62, 63, 65, 66, 67, 69, 79, 81, 85, 89, 91, 94, 100, 101, 108, 109, 115, 116, 117, 121, 123, 125, 149, 151, 158, 162, 163, 173, 175, 191, 194, 210, 220, 227, 247, 275, 295, 296, 298, 304, 307, 313, 321, 335, 344, 355, 358, 360, 374, 378, 389, 398, 402, 403, 404, 410, 413, 420, 434, 436, 438, 441, 444, 449, 451, 458, 469, 474, 475, 476, 479, 486, 493, 507, 509, 513, 515, 527, 533, 536, 539, 542, 553, 560, 561, 562, 571, 577, ...]), ([1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 25, 26, 28, 29, 31, 32, 33, 35, 36, 37, 38, 40, 41, 43, 44, 45, 46, 48, 49, 50, 51, 55, 57, 58, 59, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 76, 77, 79, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 98, 100, 101, 103, 104, 105, 106, 107, 108, 109, 110, 111, 113, 114, 115, 116, 117, 119, 120, 121, 122, 123, ...], [0, 4, 23, 24, 27, 30, 34, 39, 42, 47, 52, 53, 54, 56, 60, 75, 78, 80, 96, 97, 99, 102, 112, 118, 130, 133, 140, 147, 152, 154, 179, 181, 183, 185, 187, 189, 196, 199, 205, 208, 212, 213, 215, 217, 221, 222, 229, 231, 232, 242, 244, 245, 246, 255, 259, 270, 276, 283, 287, 289, 305, 314, 324, 325, 327, 333, 339, 343, 345, 346, 348, 357, 372, 377, 379, 380, 384, 386, 387, 390, 392, 396, 399, 401, 412, 421, 422, 432, 437, 452, 457, 460, 461, 484, 485, 488, 491, 492, 495, 500, ...]), ([0, 1, 2, 4, 5, 6, 7, 8, 10, 11, 12, 14, 15, 17, 18, 21, 22, 23, 24, 25, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 39, 41, 42, 43, 44, 46, 47, 49, 50, 52, 53, 54, 55, 56, 57, 58, 60, 62, 63, 64, 65, 66, 67, 68, 69, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 96, 97, 99, 100, 101, 102, 103, 105, 106, 107, 108, 109, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, ...], [3, 9, 13, 16, 19, 20, 26, 38, 40, 45, 48, 51, 59, 61, 70, 82, 83, 95, 98, 104, 110, 127, 129, 131, 134, 135, 136, 138, 139, 145, 157, 161, 164, 167, 172, 177, 180, 182, 184, 186, 188, 192, 193, 195, 203, 206, 228, 230, 240, 249, 251, 266, 271, 281, 284, 294, 300, 311, 312, 317, 318, 326, 332, 334, 336, 340, 350, 351, 359, 363, 364, 369, 370, 371, 381, 383, 391, 393, 405, 407, 409, 415, 416, 419, 423, 426, 427, 431, 433, 439, 440, 442, 445, 446, 450, 453, 456, 459, 463, 465, ...]), ([0, 2, 3, 4, 5, 7, 8, 9, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 23, 24, 26, 27, 28, 29, 30, 31, 34, 35, 36, 37, 38, 39, 40, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 72, 74, 75, 78, 79, 80, 81, 82, 83, 84, 85, 86, 88, 89, 90, 91, 92, 94, 95, 96, 97, 98, 99, 100, 101, 102, 104, 106, 107, 108, 109, 110, 112, 115, 116, 117, 118, 119, 121, ...], [1, 6, 10, 17, 22, 25, 32, 33, 41, 57, 71, 73, 76, 77, 87, 93, 103, 105, 111, 113, 114, 120, 124, 126, 132, 141, 142, 150, 156, 160, 166, 170, 171, 176, 178, 190, 198, 200, 202, 207, 214, 219, 224, 225, 226, 234, 237, 241, 243, 250, 252, 253, 254, 260, 262, 263, 264, 267, 268, 278, 279, 280, 282, 285, 291, 301, 310, 315, 320, 328, 337, 338, 341, 342, 347, 354, 356, 361, 362, 365, 368, 373, 375, 376, 382, 397, 400, 414, 418, 428, 429, 430, 435, 448, 454, 455, 462, 468, 480, 487, ...]), ([0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 15, 16, 17, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 30, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 45, 46, 47, 48, 51, 52, 53, 54, 56, 57, 59, 60, 61, 62, 63, 65, 66, 67, 69, 70, 71, 73, 75, 76, 77, 78, 79, 80, 81, 82, 83, 85, 87, 89, 91, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 120, 121, 123, ...], [7, 14, 18, 29, 31, 43, 44, 49, 50, 55, 58, 64, 68, 72, 74, 84, 86, 88, 90, 92, 106, 107, 119, 122, 128, 137, 143, 144, 146, 148, 153, 155, 159, 165, 168, 169, 174, 197, 201, 204, 209, 211, 216, 218, 223, 233, 235, 236, 238, 239, 248, 256, 257, 258, 261, 265, 269, 272, 273, 274, 277, 286, 288, 290, 292, 293, 297, 299, 302, 303, 306, 308, 309, 316, 319, 322, 323, 329, 330, 331, 349, 352, 353, 366, 367, 385, 388, 394, 395, 406, 408, 411, 417, 424, 425, 443, 447, 464, 466, 471, ...]), ([0, 1, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 23, 24, 25, 26, 28, 29, 33, 34, 35, 36, 37, 38, 40, 41, 43, 44, 46, 47, 48, 50, 51, 52, 54, 55, 56, 57, 59, 61, 62, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 92, 93, 94, 95, 96, 97, 98, 99, 101, 102, 103, 106, 107, 109, 110, 112, 113, 115, 116, 117, 118, 121, 123, 124, 125, 126, 130, ...], [2, 6, 22, 27, 30, 31, 32, 39, 42, 45, 49, 53, 58, 60, 63, 76, 77, 78, 91, 100, 104, 105, 108, 111, 114, 119, 120, 122, 127, 128, 129, 136, 144, 145, 162, 165, 167, 172, 175, 188, 189, 192, 197, 199, 203, 205, 217, 222, 231, 234, 236, 237, 238, 239, 241, 242, 244, 252, 264, 265, 268, 272, 273, 276, 283, 285, 288, 291, 293, 300, 308, 319, 331, 335, 343, 353, 358, 363, 366, 368, 371, 380, 393, 395, 413, 421, 424, 436, 439, 446, 449, 450, 454, 464, 468, 472, 474, 477, 482, 497, ...]), ([1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 73, 74, 76, 77, 78, 79, 80, 81, 83, 84, 85, 87, 88, 90, 91, 93, 94, 95, 96, 97, 98, 99, 100, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, ...], [0, 9, 10, 15, 33, 34, 48, 72, 75, 82, 86, 89, 92, 101, 115, 117, 135, 137, 142, 146, 154, 155, 168, 173, 177, 183, 184, 198, 202, 204, 206, 210, 213, 221, 228, 230, 245, 247, 249, 250, 251, 257, 261, 270, 277, 279, 281, 282, 295, 297, 309, 311, 315, 316, 317, 320, 321, 326, 327, 328, 337, 338, 346, 349, 350, 357, 362, 364, 373, 376, 382, 386, 389, 390, 401, 410, 411, 412, 419, 420, 422, 428, 431, 432, 434, 440, 443, 445, 447, 451, 452, 455, 456, 458, 460, 462, 470, 480, 483, 487, ...]), ([0, 2, 6, 8, 9, 10, 11, 12, 15, 16, 17, 18, 19, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 38, 39, 41, 42, 43, 44, 45, 47, 48, 49, 51, 52, 53, 54, 56, 58, 60, 62, 63, 64, 65, 67, 68, 69, 70, 71, 72, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 89, 91, 92, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 125, 127, 128, 129, 130, 131, ...], [1, 3, 4, 5, 7, 13, 14, 20, 36, 37, 40, 46, 50, 55, 57, 59, 61, 66, 73, 74, 79, 84, 88, 90, 93, 94, 95, 96, 112, 113, 124, 126, 133, 139, 141, 147, 149, 150, 151, 156, 157, 160, 161, 174, 181, 194, 215, 218, 220, 223, 227, 229, 232, 235, 253, 255, 259, 262, 267, 278, 287, 290, 292, 296, 298, 301, 303, 304, 318, 322, 323, 329, 339, 341, 342, 354, 365, 367, 369, 377, 379, 383, 384, 387, 394, 396, 397, 400, 402, 405, 406, 409, 414, 416, 429, 430, 433, 444, 453, 459, ...]), ([0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 13, 14, 15, 16, 18, 19, 20, 21, 22, 24, 26, 27, 28, 30, 31, 32, 33, 34, 36, 37, 38, 39, 40, 42, 44, 45, 46, 48, 49, 50, 51, 52, 53, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 66, 67, 69, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 103, 104, 105, 106, 108, 109, 111, 112, 113, 114, 115, 116, 117, ...], [8, 12, 17, 23, 25, 29, 35, 41, 43, 47, 54, 65, 68, 70, 85, 102, 107, 110, 118, 125, 130, 131, 132, 134, 138, 152, 158, 159, 166, 178, 179, 180, 182, 187, 190, 195, 208, 211, 214, 216, 224, 240, 243, 246, 254, 256, 260, 274, 275, 284, 289, 294, 299, 305, 307, 312, 324, 325, 330, 332, 333, 334, 336, 351, 355, 359, 361, 372, 374, 378, 388, 398, 399, 403, 408, 417, 418, 423, 427, 435, 437, 438, 442, 457, 466, 471, 485, 486, 488, 493, 494, 506, 511, 512, 513, 517, 519, 521, 524, 526, ...]), ([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 17, 20, 22, 23, 25, 27, 29, 30, 31, 32, 33, 34, 35, 36, 37, 39, 40, 41, 42, 43, 45, 46, 47, 48, 49, 50, 53, 54, 55, 57, 58, 59, 60, 61, 63, 65, 66, 68, 70, 72, 73, 74, 75, 76, 77, 78, 79, 82, 84, 85, 86, 88, 89, 90, 91, 92, 93, 94, 95, 96, 100, 101, 102, 104, 105, 107, 108, 110, 111, 112, 113, 114, 115, 117, 118, 119, 120, 122, 124, 125, 126, 127, 128, 129, 130, ...], [11, 16, 18, 19, 21, 24, 26, 28, 38, 44, 51, 52, 56, 62, 64, 67, 69, 71, 80, 81, 83, 87, 97, 98, 99, 103, 106, 109, 116, 121, 123, 140, 143, 148, 153, 163, 164, 169, 170, 171, 176, 185, 186, 191, 193, 196, 200, 201, 207, 209, 212, 219, 225, 226, 233, 248, 258, 263, 266, 269, 271, 280, 286, 302, 306, 310, 313, 314, 340, 344, 345, 347, 348, 352, 356, 360, 370, 375, 381, 385, 391, 392, 404, 407, 415, 425, 426, 441, 448, 465, 467, 469, 473, 475, 476, 479, 481, 489, 490, 491, ...])] |
| Optimization Metric | neg_mean_squared_error |
| Selected Features | Index(['MedInc', 'HouseAge', 'AveRooms', 'AveOccup', 'Latitude', 'Longitude'], dtype='object') |
| Selected Algorithm | LGBMRegressor |
| Time taken | 47.9699 |
| Selected Hyperparameters | {'boosting_type': 'gbdt', 'class_weight': None, 'colsample_bytree': 0.7952797110155084, 'importance_type': 'split', 'learning_rate': 0.1, 'max_depth': 63, 'min_child_samples': 20, 'min_child_weight': 0.001, 'min_split_gain': 0.0, 'n_estimators': 377, 'n_jobs': 16, 'num_leaves': 31, 'objective': None, 'random_state': 7, 'reg_alpha': 0, 'reg_lambda': 0, 'silent': 'warn', 'subsample': 1, 'subsample_for_bin': 200000, 'subsample_freq': 0} |
| AutoML version | 23.1.1 |
| Python version | 3.8.7 (default, Aug 25 2022, 13:59:56) \n[GCC 8.5.0 20210514 (Red Hat 8.5.0-10.1.0.1)] |
| Algorithm | #Samples | #Features | Mean Validation Score | Hyperparameters | CPU Time | Memory Usage (GB) |
|---|---|---|---|---|---|---|
| LGBMRegressor_HT | 14448 | 6 | -0.2017 | {'boosting_type': 'gbdt', 'colsample_bytree': 0.7952797110155084, 'max_depth': 63, 'n_estimators': 377, 'num_leaves': 31, 'reg_alpha': 0, 'reg_lambda': 0, 'subsample': 1} | 6.2535 | (0.0, None) |
| LGBMRegressor_HT | 14448 | 6 | -0.2017 | {'boosting_type': 'gbdt', 'colsample_bytree': 1, 'max_depth': 18, 'n_estimators': 377, 'num_leaves': 31, 'reg_alpha': 0, 'reg_lambda': 0, 'subsample': 1} | 5.6827 | (0.0, None) |
| LGBMRegressor_HT | 14448 | 6 | -0.2018 | {'boosting_type': 'gbdt', 'colsample_bytree': 1, 'max_depth': 63, 'n_estimators': 377, 'num_leaves': 31, 'reg_alpha': 1.8534128133509583e-05, 'reg_lambda': 0, 'subsample': 1} | 6.1757 | (0.0, None) |
| LGBMRegressor_HT | 14448 | 6 | -0.2018 | {'boosting_type': 'gbdt', 'colsample_bytree': 1, 'max_depth': 63, 'n_estimators': 377, 'num_leaves': 31, 'reg_alpha': 1.778506445476284e-05, 'reg_lambda': 0, 'subsample': 1} | 5.6868 | (0.0, None) |
| LGBMRegressor_HT | 14448 | 6 | -0.2018 | {'boosting_type': 'gbdt', 'colsample_bytree': 1, 'max_depth': 63, 'n_estimators': 377, 'num_leaves': 31, 'reg_alpha': 1.778412806339553e-05, 'reg_lambda': 0, 'subsample': 1} | 6.5803 | (0.0, None) |
| ... | ... | ... | ... | ... | ... | ... |
| LGBMRegressor_HT | 14448 | 6 | -0.9325 | {'boosting_type': 'dart', 'colsample_bytree': 0.4, 'max_depth': 2, 'n_estimators': 5, 'num_leaves': 7, 'reg_alpha': 1e-10, 'reg_lambda': 0.0056245538305564015, 'subsample': 0.4} | 0.1543 | (0.0, None) |
| LGBMRegressor_HT | 14448 | 6 | -0.9797 | {'boosting_type': 'dart', 'colsample_bytree': 0.7952737110155085, 'max_depth': 2, 'n_estimators': 5, 'num_leaves': 7, 'reg_alpha': 1e-10, 'reg_lambda': 1e-10, 'subsample': 0.4} | 0.1733 | (0.0, None) |
| LGBMRegressor_HT | 14448 | 6 | -0.9797 | {'boosting_type': 'dart', 'colsample_bytree': 0.7952797110155084, 'max_depth': 2, 'n_estimators': 5, 'num_leaves': 7, 'reg_alpha': 1e-10, 'reg_lambda': 1e-10, 'subsample': 0.4} | 0.1704 | (0.0, None) |
| LinearRegression_AS | 5000 | 8 | -1.2614 | {} | 0.0774 | 0.0 |
| LinearSVR_AS | 5000 | 8 | -1.6669 | {'C': 1.0} | 1.9808 | 0.0 |
We also provide the capability to visualize the results of each stage of the AutoML pipeline.
The plot below shows the scores predicted by Algorithm Selection for each algorithm. The horizontal line shows the average score across all algorithms. Algorithms below the line are colored turquoise, whereas those with a score higher than the mean are colored teal. Here we can see that the LGBMRegressor achieved the highest predicted score (orange bar), and is chosen for subsequent stages of the Pipeline.
# Each trial is a tuple of
# (algorithm, no. samples, no. features, mean CV score, hyperparameters,
# all CV scores, total CV time (s), memory usage (Gb))
trials = est1.model_selection_trials_
scores = [x[3] for x in trials]
models = [x[0] for x in trials]
y_margin = 0.10 * (max(scores) - min(scores))
s = pd.Series(scores, index=models).sort_values(ascending=False)
colors = []
for f in s.keys():
if f == '{}_AS'.format(est1.selected_model_):
colors.append('orange')
elif s[f] >= s.mean():
colors.append('teal')
else:
colors.append('turquoise')
fig, ax = plt.subplots(1)
ax.set_title("Algorithm Selection Trials")
ax.set_ylim(min(scores) - y_margin, max(scores) + y_margin)
ax.set_ylabel(est1.inferred_score_metric[0])
s.plot.bar(ax=ax, color=colors, edgecolor='black')
ax.axhline(y=s.mean(), color='black', linewidth=0.5)
plt.show()
Following Algorithm Selection, Adaptive Sampling aims to find the smallest dataset sample that can be created without compromising validation set score for the chosen model (LGBMRegressor). In this case, no subset is found to keep sufficient validation score, so we continue with the entire dataset.
# Each trial is a tuple of
# (algorithm, no. samples, no. features, mean CV score, hyperparameters,
# all CV scores, total CV time (s), memory usage (Gb))
trials = est1.adaptive_sampling_trials_
scores = [x[3] for x in trials]
n_samples = [x[1] for x in trials]
y_margin = 0.10 * (max(scores) - min(scores))
fig, ax = plt.subplots(1)
ax.set_title("Adaptive Sampling ({})".format(trials[0][0]))
ax.set_xlabel('Dataset sample size')
ax.set_ylabel(est1.inferred_score_metric[0])
ax.grid(color='g', linestyle='-', linewidth=0.1)
ax.set_ylim(min(scores) - y_margin, max(scores) + y_margin)
ax.plot(n_samples, scores, 'k:', marker="s", color='teal', markersize=3)
plt.show()
The next step of the pipeline is to find a relevant feature subset to maximize score for the chosen algorithm. AutoMLx Feature Selection follows an intelligent search strategy, looking at various possible feature rankings and subsets, and identifying that smallest feature subset that does not compromise on score for the chosen algorithm (LGBMRegressor). The orange line shows the optimal number of features chosen by Feature Selection (in this case 6 features, droppping AveBedrms and Population).
print(f"Features selected: {est1.selected_features_names_.to_list()}")
dropped_features = est1.pipeline.preprocessor.transformed_col_names.drop(est1.selected_features_names_)
print(f"Features dropped: {dropped_features.to_list()}")
# Each trial is a tuple of
# (algorithm, no. samples, no. features, mean CV score, hyperparameters,
# all CV scores, total CV time (s), memory usage (Gb))
trials = est1.feature_selection_trials_
scores = [x[3] for x in trials]
n_features = [x[2] for x in trials]
y_margin = 0.10 * (max(scores) - min(scores))
fig, ax = plt.subplots(1)
ax.set_title("Feature Selection Trials")
ax.set_xlabel("Number of Features")
ax.set_ylabel(est1.inferred_score_metric[0])
ax.grid(color='g', linestyle='-', linewidth=0.1)
ax.set_ylim(min(scores) - y_margin, max(scores) + y_margin)
ax.plot(n_features, scores, 'k:', marker="s", color='teal', markersize=3)
ax.axvline(x=len(est1.selected_features_names_), color='orange', linewidth=2.0)
plt.show()
Features selected: ['MedInc', 'HouseAge', 'AveRooms', 'AveOccup', 'Latitude', 'Longitude'] Features dropped: ['AveBedrms', 'Population']
Hyperparameter Tuning is the last stage of the AutoML pipeline, and focuses on improving the chosen algorithm's score on the reduced dataset (after Adaptive Sampling and Feature Selection). We use a novel iterative algorithm to search across many hyperparameter dimensions, and converge automatically when optimal hyperparameters are identified. Each trial in the graph below represents a particular hyperparameter configuration for the selected model.
# Each trial is a tuple of
# (algorithm, no. samples, no. features, mean CV score, hyperparameters,
# all CV scores, total CV time (s), memory usage (Gb))
trials = est1.tuning_trials_
scores = [x[3] for x in reversed(trials)]
y_margin = 0.10 * (max(scores) - min(scores))
fig, ax = plt.subplots(1)
ax.set_title("Hyperparameter Tuning Trials")
ax.set_xlabel("Iteration $n$")
ax.set_ylabel(est1.inferred_score_metric[0])
ax.grid(color='g', linestyle='-', linewidth=0.1)
ax.set_ylim(min(scores) - y_margin, max(scores) + y_margin)
ax.plot(range(1, len(trials) + 1), scores, 'k:', marker="s", color='teal', markersize=3)
plt.show()
The Oracle AutoMLx solution also supports a model_list argument, allowing you to control what algorithms AutoML considers during its optimization process. model_list is specified as a list of strings, which can be any combination of
Here, we ask AutoML to only consider LinearRegression. Note that it does not perform well compared to the full capabilites of the AutoML pipeline.
est2 = automl.Pipeline(task='regression', model_list=['LinearRegression'])
est2.fit(X_train, y_train)
y_pred = est2.predict(X_test)
score_modellist = mean_squared_error(y_test, y_pred)
print(f'Prediction error (mse) on test data : {score_modellist}')
[2023-02-10 12:51:18,957] [automl.pipeline] Random state (7) is used for model builds
[2023-02-10 12:51:18,998] [automl.pipeline] Dataset shape: (14448, 8)
[2023-02-10 12:51:19,091] [automl.pipeline] Running Auto-Preprocessing
[2023-02-10 12:51:19,117] [automl.pipeline] Preprocessing completed. Updated Dataset shape: (14448, 8), cv: [(11558, 2890), (11558, 2890), (11558, 2890), (11559, 2889), (11559, 2889), (11558, 2890), (11558, 2890), (11558, 2890), (11559, 2889), (11559, 2889)]
[2023-02-10 12:51:19,240] [automl.pipeline] SVR, KNeighborsRegressor are disabled for datasets with > 10K samples or > 1K features
[2023-02-10 12:51:19,241] [automl.pipeline] Running Model Selection
[2023-02-10 12:51:19,242] [automl.pipeline] Algorithm Selection Disabled
[2023-02-10 12:51:19,242] [automl.pipeline] User provided model LinearRegression is the only desired model
[2023-02-10 12:51:19,273] [automl.pipeline] Model Selection completed. Selected model: ['LinearRegression']
[2023-02-10 12:51:19,274] [automl.pipeline] Running Adaptive Sampling. Dataset Shape: (14448, 8), Valid Shape: None, CV: [(11558, 2890), (11558, 2890), (11558, 2890), (11559, 2889), (11559, 2889), (11558, 2890), (11558, 2890), (11558, 2890), (11559, 2889), (11559, 2889)], Class counts: N/A
[2023-02-10 12:51:27,204] [automl.pipeline] Adaptive Sampling Completed. Updated Dataset Shape: (14448, 8), Valid Shape: None, CV: [(4343, 2890), (4343, 2890), (4343, 2890), (4343, 2889), (4343, 2889), (4343, 2890), (4343, 2890), (4343, 2890), (4343, 2889), (4343, 2889)], Class counts: N/A
[2023-02-10 12:51:27,205] [automl.pipeline] Starting Feature Selection 0. Dataset Shape: (14448, 8)
[2023-02-10 12:51:30,795] [automl.pipeline] Feature Selection 0 completed. Updated Dataset shape: (14448, 4)
[2023-02-10 12:51:30,866] [automl.pipeline] Tuning LinearRegression
[2023-02-10 12:51:30,999] [automl.pipeline] Tuning completed. Best params: {}
[2023-02-10 12:51:31,127] [automl.pipeline] (Re)fitting Pipeline
[2023-02-10 12:51:31,582] [automl.xengine] Local ProcessPool execution (n_jobs=40)
[2023-02-10 12:51:31,667] [automl.pipeline] AutoML completed. Time taken - 12.239 sec
Prediction error (mse) on test data : 0.5530449940668276
You can specify a custom validation set that you want AutoML to use to evaluate the quality of models and configurations.
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, train_size=0.7, random_state=0)
est3 = automl.Pipeline(task='regression')
est3.fit(X_train, y_train, X_val, y_val)
y_pred = est3.predict(X_test)
score = mean_squared_error(y_test, y_pred)
print(f'Prediction error (MSE) on test data : {score}')
[2023-02-10 12:51:35,488] [automl.pipeline] Random state (7) is used for model builds
[2023-02-10 12:51:35,524] [automl.pipeline] Dataset shape: (10113, 8)
[2023-02-10 12:51:35,605] [automl.pipeline] Running Auto-Preprocessing
[2023-02-10 12:51:35,633] [automl.pipeline] Preprocessing completed. Updated Dataset shape: (10113, 8), cv: None
[2023-02-10 12:51:35,754] [automl.pipeline] SVR, KNeighborsRegressor are disabled for datasets with > 10K samples or > 1K features
[2023-02-10 12:51:35,755] [automl.pipeline] Running Model Selection
[2023-02-10 12:51:43,486] [automl.pipeline] Model Selection completed. Selected model: ['LGBMRegressor']
[2023-02-10 12:51:43,487] [automl.pipeline] Running Adaptive Sampling. Dataset Shape: (10113, 8), Valid Shape: (4335, 8), CV: None, Class counts: N/A
[2023-02-10 12:51:47,008] [automl.pipeline] Adaptive Sampling Completed. Updated Dataset Shape: (10113, 8), Valid Shape: (1667, 8), CV: None, Class counts: N/A
[2023-02-10 12:51:47,009] [automl.pipeline] Starting Feature Selection 0. Dataset Shape: (10113, 8)
[2023-02-10 12:51:48,970] [automl.pipeline] Feature Selection 0 completed. Updated Dataset shape: (10113, 7)
[2023-02-10 12:51:49,041] [automl.pipeline] Tuning LGBMRegressor
[2023-02-10 12:52:00,959] [automl.pipeline] Tuning completed. Best params: {'boosting_type': 'gbdt', 'colsample_bytree': 1, 'max_depth': 17, 'n_estimators': 376, 'num_leaves': 31, 'reg_alpha': 0, 'reg_lambda': 0, 'subsample': 1}
[2023-02-10 12:52:01,094] [automl.pipeline] (Re)fitting Pipeline
[2023-02-10 12:52:01,953] [automl.xengine] Local ProcessPool execution (n_jobs=40)
[2023-02-10 12:52:02,042] [automl.pipeline] AutoML completed. Time taken - 26.076 sec
Prediction error (MSE) on test data : 0.20439839891710868
Similarly the user can control the Feature Selection stage of the AutoML Pipeline. This can be achieved by using the min_features argument when initializing the Pipeline. Setting it to 1.0 will disable Feature selection.
est4 = automl.Pipeline(task='regression', min_features=1.0)
est4.fit(X_train, y_train, X_val, y_val)
y_pred = est4.predict(X_test)
score = mean_squared_error(y_test, y_pred)
print(f'Prediction error (MSE) on test data : {score}')
[2023-02-10 12:52:02,415] [automl.pipeline] Random state (7) is used for model builds
[2023-02-10 12:52:02,450] [automl.pipeline] Dataset shape: (10113, 8)
[2023-02-10 12:52:02,535] [automl.pipeline] Running Auto-Preprocessing
[2023-02-10 12:52:02,563] [automl.pipeline] Preprocessing completed. Updated Dataset shape: (10113, 8), cv: None
[2023-02-10 12:52:02,694] [automl.pipeline] SVR, KNeighborsRegressor are disabled for datasets with > 10K samples or > 1K features
[2023-02-10 12:52:02,695] [automl.pipeline] Running Model Selection
[2023-02-10 12:52:11,935] [automl.pipeline] Model Selection completed. Selected model: ['LGBMRegressor']
[2023-02-10 12:52:11,936] [automl.pipeline] Running Adaptive Sampling. Dataset Shape: (10113, 8), Valid Shape: (4335, 8), CV: None, Class counts: N/A
[2023-02-10 12:52:15,515] [automl.pipeline] Adaptive Sampling Completed. Updated Dataset Shape: (10113, 8), Valid Shape: (1667, 8), CV: None, Class counts: N/A
[2023-02-10 12:52:15,516] [automl.pipeline] Starting Feature Selection 0. Dataset Shape: (10113, 8)
[2023-02-10 12:52:15,589] [automl.pipeline] Feature Selection Disabled
[2023-02-10 12:52:15,590] [automl.pipeline] Using all features: Index(['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup',
'Latitude', 'Longitude'],
dtype='object')
[2023-02-10 12:52:15,614] [automl.pipeline] Feature Selection 0 completed. Updated Dataset shape: (10113, 8)
[2023-02-10 12:52:15,686] [automl.pipeline] Tuning LGBMRegressor
[2023-02-10 12:52:21,509] [automl.pipeline] Tuning completed. Best params: {'boosting_type': 'gbdt', 'colsample_bytree': 0.7952737110155085, 'max_depth': 63, 'n_estimators': 100, 'num_leaves': 377, 'reg_alpha': 0, 'reg_lambda': 0, 'subsample': 1}
[2023-02-10 12:52:21,643] [automl.pipeline] (Re)fitting Pipeline
[2023-02-10 12:52:23,035] [automl.xengine] Local ProcessPool execution (n_jobs=40)
[2023-02-10 12:52:23,128] [automl.pipeline] AutoML completed. Time taken - 20.233 sec
Prediction error (MSE) on test data : 0.20085671762850343
The Oracle AutoMLx tool tries to maximize a given scoring metric, by looking at different algorithms, features and hyperparameter choices. By default, the score metric is set to neg_mean_squared_error for regression.
The user can also choose a metric using score_metric. The scoring metric can be specified as a string. For regression, supported metrics are
Detailed documentation about scoring metrics: https://scikit-learn.org/stable/modules/classes.html#regression-metrics.
Here we ask AutoML to optimize for the r2 scoring metric.
est5 = automl.Pipeline(task='regression', score_metric='r2')
est5.fit(X_train, y_train, X_val, y_val)
y_pred = est5.predict(X_test)
score_scoringstr = mean_squared_error(y_test, y_pred)
print(f'Prediction error (MSE) on test data : {score_scoringstr}')
[2023-02-10 12:52:24,004] [automl.pipeline] Random state (7) is used for model builds
[2023-02-10 12:52:24,039] [automl.pipeline] Dataset shape: (10113, 8)
[2023-02-10 12:52:24,126] [automl.pipeline] Running Auto-Preprocessing
[2023-02-10 12:52:24,160] [automl.pipeline] Preprocessing completed. Updated Dataset shape: (10113, 8), cv: None
[2023-02-10 12:52:24,293] [automl.pipeline] SVR, KNeighborsRegressor are disabled for datasets with > 10K samples or > 1K features
[2023-02-10 12:52:24,295] [automl.pipeline] Running Model Selection
[2023-02-10 12:52:34,393] [automl.pipeline] Model Selection completed. Selected model: ['LGBMRegressor']
[2023-02-10 12:52:34,395] [automl.pipeline] Running Adaptive Sampling. Dataset Shape: (10113, 8), Valid Shape: (4335, 8), CV: None, Class counts: N/A
[2023-02-10 12:52:37,007] [automl.pipeline] Adaptive Sampling Completed. Updated Dataset Shape: (6367, 8), Valid Shape: (1667, 8), CV: None, Class counts: N/A
[2023-02-10 12:52:37,008] [automl.pipeline] Starting Feature Selection 0. Dataset Shape: (6367, 8)
[2023-02-10 12:52:39,047] [automl.pipeline] Feature Selection 0 completed. Updated Dataset shape: (6367, 5)
[2023-02-10 12:52:39,118] [automl.pipeline] Tuning LGBMRegressor
[2023-02-10 12:52:42,835] [automl.pipeline] Tuning completed. Best params: {'boosting_type': 'gbdt', 'colsample_bytree': 1, 'max_depth': 18, 'n_estimators': 252, 'num_leaves': 31, 'reg_alpha': 0, 'reg_lambda': 0, 'subsample': 1}
[2023-02-10 12:52:42,917] [automl.pipeline] (Re)fitting Pipeline
[2023-02-10 12:52:43,651] [automl.xengine] Local ProcessPool execution (n_jobs=40)
[2023-02-10 12:52:43,740] [automl.pipeline] AutoML completed. Time taken - 19.258 sec
Prediction error (MSE) on test data : 0.2122826811540096
For a variety of decision-making tasks, getting only a prediction as model output is not sufficient. A user may wish to know why the model outputs that prediction, or which data features are relevant for that prediction. For that purpose the Oracle AutoMLx solution defines the MLExplainer object, which allows to compute multiple types of model explanations.
The MLExplainer object takes as argument the trained model, the training data and labels, as well as the task.
explainer = automl.MLExplainer(est1,
X_train,
y_train,
target_names=['Median Price'],
task="regression")
The notion of Global Feature Importance intuitively measures how much the model's performance (relative to the provided train labels) would change if a given feature were dropped from the dataset, without allowing the model to be retrained. This notion of feature importance considers each feature independently from all other features.
By default we use a permutation method to successively measure the importance of each feature. Such a method therefore runs in linear time with respect to the number of features in the dataset.
The method explain_model() allows to compute such feature importances. It also provides 95% confidence intervals for each feature importance.
result_explain_model_default = explainer.explain_model()
There are two options to show the explanation's results:
to_dataframe() will return a dataframe of the results.show_in_notebook() will show the results as a bar plot.The features are returned in decreasing order of importance. We see that Latitude and Longitude are considered to be the most important features.
result_explain_model_default.to_dataframe()
| feature | attribution | upper_bound | lower_bound | |
|---|---|---|---|---|
| 0 | Latitude | 1.159111 | 1.187185 | 1.131037 |
| 1 | Longitude | 1.025946 | 1.059236 | 0.992655 |
| 2 | MedInc | 0.337256 | 0.341957 | 0.332555 |
| 3 | AveOccup | 0.168229 | 0.172627 | 0.163831 |
| 4 | AveRooms | 0.122563 | 0.125429 | 0.119696 |
| 5 | HouseAge | 0.058460 | 0.062614 | 0.054306 |
result_explain_model_default.show_in_notebook()
Another way to measure dependency on a feature is through a partial dependence plot (PDP) or an individual conditional expectation (ICE) plot. For accumulated local effects (ALE) explanations, see Advanced Feature Dependence Options (ALE)
Given a dataset, a PDP displays the average output of the model as a function of the value of the selected set of features.
It can be computed for a single feature, as in the cell below. The X-axis is the value of the Latitude feature and the y-axis is the corresponding outputted price. Since we are considering the whole dataset, the average over outputs is given by the red line, while the shaded interval corresponds to a 95% confidence interval for the average.
The histogram on top shows the distribution of the value of the Latitude feature in the dataset.
result_explain_feature_dependence_default = explainer.explain_feature_dependence('Latitude')
result_explain_feature_dependence_default.show_in_notebook()
The ICE plot is automatically computed at the same time as any one-feature PDP. It can be accessed by passing ice=True to show_in_notebook.
Similar to PDPs, ICE plots show the median prediction as a model red line. However, the variance in the model's predictions are shown by plotting the predictions of a sample of individual data instances as light grey lines. (For categorical features, the distribution in the predictions is instead shown as a violin plot.)
result_explain_feature_dependence_default.show_in_notebook(ice=True)
We can also plot the PDP for multiple features. The plot below is the PDP for the Latitude and Longitude features. The X-axis still shows the values of Latitude, while there is a different curve and confidence interval for each value of the feature Longitude.
The histogram displays the joint distribution of the two features.
result_explain_feature_dependence_default = explainer.explain_feature_dependence(['Latitude', 'Longitude'])
result_explain_feature_dependence_default.show_in_notebook()
PDPs can be computed for an arbitrary number of variables; however, they can only be visualized with up to 4. Here, we show an example with 3 features.
result_explain_feature_dependence_default = explainer.explain_feature_dependence(['MedInc', 'HouseAge', 'AveOccup'])
result_explain_feature_dependence_default.show_in_notebook()
Given a data sample, one can also obtain the local importance, which is the importance of the features for the model's prediction on that sample.
In the following cell, we consider sample $0$. The function explain_prediction() computes the local importance for a given sample.
Latitude=34.02 means that the value of feature Latitude for that sample is 34.02. Removing that feature would change the model's prediction by the magnitude of the bar. That is, in this case, the model's prediction for the median house price is approximately 1.15 (i.e., $115 000) less because the model knows the value of Latitude is 34.02.
index = 0
result_explain_prediction_default = explainer.explain_prediction(X_train.iloc[index:index + 1, :])
result_explain_prediction_default[0].show_in_notebook()
We now summarize all of the individual local feature importance explanations into one single aggregate explanation.
# We select 10 random instances here as an example and show the aggregate explanation of those instances.
local_explanations = explainer.explain_prediction(X_train.sample(n=10))
alfi = explainer.aggregate(explanations=local_explanations)
alfi.show_in_notebook()
The Oracle AutoMLx solution offers also What-IF tool to explain a trained ML model's predictions through a simple interactive interface.
You can use What-IF explainer to explore and visualize immediately how changing a sample value will affect the model's prediction. Furthermore, What-IF can be used to visualize how model's predictions are related to any feature of the dataset.
explainer.explore_whatif(X_test, y_test)
We now display more advanced configuration for computing feature importance. Here, we will explain a custom LinearRegression estimator from scikit-learn. Note that the MLExplainer object is capable of explaining any regression model, as long as the model follows a scikit-learn-style interface with the predict function.
scikit_model = LinearRegression()
scikit_model.fit(X_train, y_train)
explainer_sklearn = automl.MLExplainer(scikit_model,
X_train,
y_train,
target_names=['Median Price'],
task="regression")
One can modify the number of iterations n_iter used to evaluate the global importance of the model, or the local importance of a prediction.
Increasing n_iter requires a linear increase in computation time. It however provides more
accurate importance estimates, thereby decreasing the variance in repeated calls to
explain_model/explain_prediction.
The default value is auto, which selects a suitable default value based on the choice of
the method of explanation.
result_explain_model_increase_n_iter = explainer_sklearn.explain_model(n_iter=20)
result_explain_model_increase_n_iter.show_in_notebook()
The Oracle AutoMLx solution allows one to change the effect of feature interactions. This can be done through the tabulator_type argument of both global and local importance methods.
tabulator_type can be set to one of the following options:
permutation: This value is the default method in the MLExplainer object, with the behaviour described above
shapley: Feature importance is computed using the popular game-theoretic Shapley value method. Technically, this measures the importance of each feature while including the effect of all feature interactions. As a result, it runs in exponential time with respect to the number of features in the dataset. This method also includes the interaction effects of the other features, which means that if two features contain duplicate information, they will be less important. Note that the interpretation of this method's result is a bit different from the permutation method's result. An interested reader may find this a good source for learning more about it.
kernel_shap: Feature importance attributions will be calculated using an approximation of the Shapley value method. It typically provides relatively high-quality approximations; however, it currently does not provide confidence intervals.
shap_pi: Feature importance attributions will be computed using an approximation of the Shapley value method. It runs in linear time, but may miss the effect of interactions between some features, which may therefore produce lower-quality results. Most likely, you will notice that this method yields larger confidence intervals than the other two.
Summary: permutation can miss important features for AD. Exact SHAP (shapley) doesn't, but it is exponential. kernel_shap is an approximation of exact SHAP method that does not provide confidence intervals. shap_pi is linear, thus faster than exact SHAP and kernel_shap but unstable and very random leads to lower quality approximations.
explainer_sklearn.configure_explain_prediction(tabulator_type="kernel_shap")
index = 0
result_explain_prediction_kernel_shap = explainer_sklearn.explain_prediction(X_train.iloc[index:index+1, :])
result_explain_prediction_kernel_shap[0].show_in_notebook()
Oracle AutoMLx solution also provides the evaluator_type attribute, which allows one to choose whether to get feature importance attributions that explain exactly which features the model has learned to use (interventional), or which features the model or a retrained model could have learned to use (observational).
interventional : The computed feature importances are as faithful to the
model as possible. That is, features that are ignored by
the model will not be considered important. This setting
should be preferred if the primary goal is to learn about
the machine learning model itself. Technically, this
setting is called 'interventional', because the method will
intervene on the data distribution when assessing the
importance of features. The intuition of feature importance attributions computed with this method is that the features are dropped from the dataset and the model is not allowed to retrain.
observational : The computed feature importances are more faithful to
the relationships that exist in the real world (i.e., relationships
observed in the dataset), even if your specific model did not learn
to use them. For example, when using a permutation tabulator, a feature
that is used by the model will not show a large impact on the model's
performance if there is a second feature that contains near-duplicate
information, because a re-trained model could have learned to use the
other feature instead. (Similarly, for Shapley-based tabulators, a
feature that is ignored by the model may have a non-zero feature
importance if it could have been used by the model to
predict the target.) This setting should be preferred if the
model is merely a means to learn more about the
relationships that exist within the data. Technically, this
setting is called 'observational', because it observes the
relationships in the data without breaking the existing
data distribution.
explainer_sklearn.configure_explain_model(evaluator_type="observational")
result_explain_model_kernel_shap = explainer_sklearn.explain_model()
result_explain_model_kernel_shap.show_in_notebook()
explainer_sklearn.configure_explain_prediction(evaluator_type="observational",
tabulator_type="permutation")
index = 0
result_explain_prediction_kernel_shap = explainer_sklearn.explain_prediction(X_train.iloc[index:index+1, :])
result_explain_prediction_kernel_shap[0].show_in_notebook()
The Oracle AutoMLx solution allows one to change the type of local explainer effect of feature interactions. This can be done through the explainer_type argument of local importance methods.
explainer_type can be set to one of the following options:
perturbation: This value is the default explainer type in local feature importance. As we showed above, the explanation(s) will be computed by perturbing the features of the indicated data instance(s) and measuring the impact on the model predictions.
surrogate: The LIME-style explanation(s) will be computed by fitting a surrogate model to the predictions of the original model in a small region around the indicated data instance(s) and measuring the importance of the features to the interpretable surrogate model. The method of surrogate explainer can be set to one of the following options:
systematic: An Oracle-labs-improved version of LIME that uses a systematic sampling and custom sample weighting. (Default)lime: Local interpretable model-agnostic explanations (LIME) algorithm (https://arxiv.org/pdf/1602.04938).explainer_sklearn.configure_explain_prediction(explainer_type='surrogate')
[2023-02-10 12:53:23,553] [automl.mlx] AutoMLx got an unexpected keyword argument 'evaluator_type', which is not a configurable attribute of any of ['TabularLocalSurrogateExplainer', 'SystematicSampleGeneration', 'SiloWeighting', 'SurrogateHandler'].
Valid options are:
{'TabularLocalSurrogateExplainer': ['method', 'exp_sorting', 'num_features', 'scale_weight'], 'SystematicSampleGeneration': ['discretizer', 'num_samples', 'n_bins'], 'SiloWeighting': [], 'SurrogateHandler': ['model', 'feature_selection', 'force_fit_sample']}
index = 0
result_explain_prediction_surrogate_lime = explainer_sklearn.explain_prediction(X_train.iloc[index:index+1, :])
result_explain_prediction_surrogate_lime[0].show_in_notebook()
explainer_sklearn.configure_explain_prediction(explainer_type='surrogate',
method='systematic')
[2023-02-10 12:53:30,919] [automl.mlx] AutoMLx got an unexpected keyword argument 'evaluator_type', which is not a configurable attribute of any of ['TabularLocalSurrogateExplainer', 'SystematicSampleGeneration', 'SiloWeighting', 'SurrogateHandler'].
Valid options are:
{'TabularLocalSurrogateExplainer': ['method', 'exp_sorting', 'num_features', 'scale_weight'], 'SystematicSampleGeneration': ['discretizer', 'num_samples', 'n_bins'], 'SiloWeighting': [], 'SurrogateHandler': ['model', 'feature_selection', 'force_fit_sample']}
index = 0
result_explain_prediction_surrogate_systematic = explainer_sklearn.explain_prediction(X_train.iloc[index:index + 1, :])
result_explain_prediction_surrogate_systematic[0].show_in_notebook()
We now show how to use an alternative method for computing feature dependence: accumulated local effects (ALE). ALE explanations are sometimes considered a better alternative to PDPs when features are correlated, because it does not evaluate the model outside of its training distribution in these cases. For more information, see https://christophm.github.io/interpretable-ml-book/ale.html.
Given a dataset, an ALE displays the average change in the output of the model, accumulated over multiple small changes in one or two features, when all other features are held fixed. By default, the ALE explanations are centered around 0, and thus, unlike PDPs, ALEs show the change in the prediction measured by changing a given feature, rather than the average model's prediction for a particular feature value.
It can be computed for a single feature, as in the cell below. The X-axis is the value of the Latitude feature and the y-axis is the corresponding computed ALE (price unit).
The histogram on top shows the distribution of the value of the Latitude feature in the dataset.
explainer_sklearn.configure_explain_feature_dependence(explanation_type='ale')
result_explain_feature_dependence_default = explainer_sklearn.explain_feature_dependence('Latitude')
result_explain_feature_dependence_default.show_in_notebook()
We can also compute ALEs for two features (At least one is numerical). The plot below is the ALE plot for the Latitude and Longitude features. The X-axis still shows the values of Latitude, while there is multiple lines, one for each value of the feature Longitude.
The histogram displays the joint distribution of the two features.
result_explain_feature_dependence_default = explainer_sklearn.explain_feature_dependence(['Latitude', 'Longitude'])
result_explain_feature_dependence_default.show_in_notebook()